Image retrieval using automatic region tagging

The task of tagging, annotating or labelling image content automatically with semantic keywords is a challenging problem. To automatically tag images semantically based on the objects that they contain is essential for image retrieval. In addressing these problems, we explore the techniques developed to combine textual description of images with visual features, automatic region tagging and region-based ontology image retrieval. To evaluate the techniques, we use three corpora comprising: Lonely Planet travel guide articles with images, Wikipedia articles with images and Goats comic strips. In searching for similar images or textual information specified in a query, we explore the unification of textual descriptions and visual features (such as colour and texture) of the images. We compare the effectiveness of using different retrieval similarity measures for the textual component. We also analyse the effectiveness of different visual features extracted from the images. We then investigate the best weight combination of using textual and visual features. Using the queries from the Multimedia Track of INEX 2005 and 2006, we found that the best weight combination significantly improves the effectiveness of the retrieval system. Our findings suggest that image regions are better in capturing the semantics, since we can identify specific regions of interest in an image. In this context, we develop a technique to tag image regions with high-level semantics. This is done by combining several shape feature descriptors and colour, using an equal-weight linear combination. We experimentally compare this technique with more complex machine-learning algorithms, and show that the equal-weight linear combination of shape features is simpler and at least as effective as using a machine learning algorithm. We focus on the synergy between ontology and image annotations with the aim of reducing the gap between image features and high-level semantics. Ontologies ease information retrieval. They are used to mine, interpret, and organise knowledge. An ontology may be seen as a knowledge base that can be used to improve the image retrieval process, and conversely keywords obtained from automatic tagging of image regions may be useful for creating an ontology. We engineer an ontology that surrogates concepts derived from image feature descriptors. We test the usability of the constructed ontology by querying the ontology via the Visual Ontology Query Interface, which has a formally specified grammar known as the Visual Ontology Query Language. We show that synergy between ontology and image annotations is possible and this method can reduce the gap between image features and high-level semantics by providing the relationships between objects in the image. In this thesis, we conclude that suitable techniques for image retrieval include fusing text accompanying the images with visual features, automatic region tagging and using an ontology to enrich the semantic meaning of the tagged image regions

Content-based image retrieval using image regions as query examples

A common approach to content-based image retrieval is to use example images as queries; images in the collection that have low-level features similar to the query examples are returned in response to the query. In this paper, we explore the use of image regions as query examples. We compare the retrieval e ectiveness of using whole images, single regions, and multiple regions as examples. We also compare two approaches for combining shape features: an equalweight linear combination, and classi cation using machine learning algorithms. We show that using image regions as query examples leads to higher e ectiveness than using whole images, and that an equalweight linear combination of shape features is simpler and at least as e ective as using a machine learning algorithm

A visual ontology query interface of content-based image retrieval

Not availabl

Social Media Retrieval Using Image Features and Structured Text

This paper presents an experimental study that examines the performance of various combination techniques for content-based image retrieval using a fusion of visual and textual search results. The evaluation is comprehensively benchmarked using more than 160,000 samples from INEX-MM2006 images dataset and the corresponding XML documents. For visual search, we have successfully combined Hough transform, Object's color histogram, and Texture (H.O.T). For comparison purposes, we used the provided UvA features. Based on the evaluation, our submissions show that Uva+Text combination performs most effectively, but it is closely followed by our H.O.T- (visual only) feature. Moreover, H.O.T+Text performance is still better than UvA (visual)only. These findings show that the combination of effective text and visual search results can improve the overall performance of CBIR in Wikipedia collections which contain a heterogeneous (i.e. wide) range of genres and topics

Effects of relevance criteria and subjective factors on web image searching behaviour

Searching for images is an everyday activity. Nevertheless, even a highly skilled searcher often struggles to find what they are looking for. This article studies the factors that affect users' online web image search behaviour, investigating (1) the use of criteria in making image relevance judgements and (2) the effect of familiarity, difficulty and satisfaction. The study includes 48 users who performed four online image search tasks using Google Images. Simulated work scenarios, questionnaires and screen capture recordings were used to collect data of their image search behaviour. The results show in judging image relevance, users may apply similar criterion, however, the importance of these criteria depends on the type of image search. Similarly, ratings of users' perception on subjective aspects of performing image search shows they were task dependent. Users' perception on subjective aspects of performing image search did not always correspond with their actual search behaviour. Correlation analysis shows that subjective factors cannot be definitively measured by using only one component of search behaviour. Future work includes further analysis on the effects of topic familiarity and satisfaction

Combining image and structured text retrieval

Two common approaches in retrieving images from a collection are retrieval by text keywords and retrieval by visual content. However, it is widely recognised that it is impossible for keywords alone to fully describe visual content. This paper reports on the participation of the RMIT University group in the INEX 2005 multimedia track, where we investigated our approach of combining evidence from a content-oriented XML retrieval system and a content-based image retrieval system using a linear combination of evidence. Our approach yielded the best overall result for the INEX 2005 Multimedia track using the standard evaluation measures. We have extended our work by varying the parameter for the linear combination of evidence, and we have also examined the performance of runs submitted by participants by using the newly proposed HiXEval evaluation metric. We show that using CBIR in conjunction with text search leads to better retrieval performance